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TITLE OF THE INVENTION 
IMAGE SEARCH METHOD AND APPARATUS 

FIELD OF THE INVENTION 
The present invention relates to an image search 
method and apparatus for searching for desired image 
data from a plurality of image data, and a medium. 



Various means for searching for a desired image 
from an image database in which a plurality of image 
data are stored have been proposed. Search processing 
methods of this type are roughly classified into/ 



* methods of linking nonimage information such as 
a keyword or image sensing date and time to image data 
and executing search on the basis of it, and 

• methods of executing search on the basis of a 
feature amount (information such as luminance/color 
difference information, image frequency, or histogram) 
of an image itself 

In the latter methods, a method of presenting an 
image to an image database and searching for an image 
using the feature amount of the image as a search key 
is particularly called similar image search. It has an 
advantage that an intuitively understandable search 
interface can be provided to a user who has no special 



BACKGROUND OF THE INVENTION 



knowledge about image processing. 

This similar image search is sometimes executed 
on the basis of, e.g., the user 1 s memory or an image 
idea uniquely presented by the user. Such a method is 
5 particularly effective when the user has no image data 
serving as a search key. However, similar image search 
based on such a manually drawn image is difficult with 
the above-described user interface of similar image 
search . 

10 The user is also forced to do^cumbersome 

: sj operation of, e.g., selecting a candidate image by the 

= s 

III above-described keyword search and executing similar 

C3 

image search using the image obtained by this method as 
a key image. 

15 In addition, in executing similar image search by 

"manually" drawing an image that the user remembers and 
using this image as a key image, a problem arises due 
to the fact that the human memory is ambiguous. 
Especially, for "colors", a person often remembers only 
20 characteristic colors, and also often remembers colors 
such as "red" close to primary colors. 

For this reason, even when the colors (RGB) of a 
manually drawn image are directly used as image search 
keys, it is often difficult to search for an image 
25 desired by the user. That is, in executing search 

using a manually drawn image as an original image, the 
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user interface of similar image search is hard to use 
or unusable. 

SUMMARY OF THE INVENTION 
5 The present invention has been made in 

consideration of the above-described problems, and has 
as its object to enable similar image search while 
accurately reflecting the user's intention. 

In order to achieve the above object, according 

10 to the present invention, there is provided an image 
search method of searching for a desired image from a 
plurality of images stored in storage means, comprising 
the setting step of setting a weight value in 
correspondence with a property of feature amount used 

15 in similarity calculation of the image, the calculation 
step of calculating similarity between a designated 
search source image and each of the plurality of images 
on the basis of a feature amount of the designated 
search source image, a feature amount of each of the 

20 plurality of images, and the weight value set in the 

setting step, and the acquisition step of acquiring an 
image as a search result from the plurality of images 
on the basis of the image similarity calculated in the 
calculation step. 

25 According to the present invention, there is also 

provided an image search apparatus for searching for a 
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desired image from a plurality of images stored in 
storage means, comprising setting means for setting a 
weight value in correspondence with a property of 
feature amount used in similarity calculation of the 
5 image, calculation means for calculating similarity 
between a designated search source image and each of 
the plurality of images on the basis of a feature 
amount of the designated search source image, a feature 
amount of each of the plurality of images, and the 
10 weight value set by the setting means, and acquisition 
means for acquiring an image as a search result from 
the plurality of images on the basis of the image 
similarity calculated by the calculation means. 

Other features and advantages of the present 
15 invention will be apparent from the following 
^ description taken in conjunction with the accompanying 

drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
25 with the description, serve to explain the principles 
of the invention. 
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Fig. 1 is a block diagram showing the arrangement 
of an image search apparatus according to the first 
embodiment ; 

Fig. 2 is a - fluw uharL * showing the schematic flow 
of image search according to the first embodiment; 

Fig. 3 is a view showing an operation window 
displayed on a display section 103 in step S21; 

Fig. 4 is a view for explaining an image 
segmented state in the first embodiment; 



:low chart. 



10 Fig. 5 is a "flow chact; for explaining image 

feature amount calculation processing according to the 
first embodiment; 

Fig. 6 is a ■ felow chart for explaining a method of 
calculating the average values of the R, G, and B 
15 values in each region; 

y Fig. 7 is a flow — chart* showing inter-image 

distance calculation processing according to the first 
embodiment; / y 

Fig. 8 is a flow chart ' showing processing of 
20 selecting a similar image; 

Fig. 9 is a view showing an operation window 
displayed on the display section 103 in step S24; 

Fig. 10 is a view for explaining a window 
structure for interactive image presentation in the 
2 5 second embodiment; 

Fig. 11 is a « flow ohcrrt - showing weight setting 
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processing according to the second, embodiment ; 

Fig. 12 is a view showing an operation window 
according to the third embodiment; 
/9 Fig. 13 is a ■ flow chart showing inter-image 

5 distance calculation processing according to the third 
embodiment; and 

Fig. 14 is a view for explaining a window 
structure for interactive image presentation in the 
fourth embodiment . 



10 

rj DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

in Preferred embodiments of the present invention 

%M 

£ Jj 

will now be described in detail in accordance with the 

5 

f; accompanying drawings. 

^ 15 

O [First Embodiment] 

Fig. 1 is a block diagram showing the arrangement 
of an image search apparatus (computer system) 
according to the first embodiment. 

20 Referring to Fig. 1, a CPU 101 controls the 

entire system. A keyboard 102 is used to input a 
user's instruction operation to the system together 
with a mouse 102a. A display section 103 is 
constructed by a CRT or liquid crystal panel. A ROM 

25 104 and RAM 105 constitute the storage device of the. 
system and store a program to be executed by the CPU 
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101 or data to be used by the system. A hard disk 
device 106 and floppy disk device 107 constitute an 
external storage device used by the file system of the 
system. Image data as a target of search processing 
(to be described later) is stored in this external 
storage device. Reference numeral 108 denotes a 
printer. 

Fig. 2 is a flow chart showing the schematic flow 
of image search according to the first embodiment. In 
step S21, the user draws an illustration similar to the 
desired image on the display section 103. In step S22, 
the feature amount of the drawn image is calculated. 
In step S23, a similar image is searched on the basis 
of the feature amount. In step S24, similar image data 
obtained by the search is displayed on the display 
section 103. Each of these steps will be described 
below in detail. 
<Description of Step S21> 

Fig. 3 is a view showing an operation window 
displayed on the display section 103 in step S21. 
Reference numeral 31 denotes a user drawing region; 32, 
color designation scroll bars; 33, a clear button; 34, 
an undo button; 35, radio buttons used to designate the 
line width of a pen; 36, tool pallet buttons; 37, a 
search execution button, and 38, a set button used to 
set a condition for the search. 



The user can draw an illustration similar to the 
image to be searched in the user drawing region 31 
using the mouse 102a, the drawing tools realized by 
software, and the like. The outline of operations of 
5 the drawing tools used to draw an illustration will be 
described below . 

The color designation scroll bars 32 are used to 
designate the color of a pen to be used for drawing. R, 
W G, and B values are designated sequentially from the 

"S 5 10 upper side. When the clear button 33 is pressed, the 

entire drawing region 31 is made white to initialize 

O 

10 the drawing region 31. When the undo button 34 is 

s pressed, the immediately preceding drawing operation is 

f-i. canceled to restore the previous state. The line width 

15 of the pen to be used for drawing in the drawing region 
;q! 31 can be designated using the radio buttons 35. When • 

the left "pen" button of the tool pallet 36 is selected, 
the user can draw a free line on the drawing region 31 
using the pointing device 102a. When the central 
20 "straight line" button of the tool pallet 36 is 

selected, a straight line can be drawn by designating 
the start and end points of the straight line using the 
pointing device 102a. When the right "circle" button 
of the tool pallet 36 is selected, a circle can be 
25 drawn by designating the center and radius of the 
circle, like the "pen" or "straight line" button. 
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When an illustration is drawn in the drawing 
region 31 using the above drawing tools, and the search 
execution button 37 is pressed, processing advances 
from step S21 to step S22. 

Before pressing the search execution button 37, a 
similar image search condition for processing from step 
S22 can be changed by operating the set button 38. 
More specifically, when the user presses the set button 
38, a setting menu 39 is displayed in which, for 
calculation of similarity in similar image search, "a 
uniform weight for the entire image (entire image) " or 
"an increased weight for the central portion of the 
image (central portion)" can be selected. 
<Description of Step S22> 

In step S22, the image feature amount of the 
illustration is calculated. In this embodiment, the 
image is segmented into a predetermined number of 
regions, feature amounts are calculated in units of 
regions, and the similarity is calculated using these 
feature amounts. Fig. 4 is a view for explaining an 
image segmented state in the first embodiment. As 
shown in Fig. 4, the size of the drawing region 31 is W 
pixels (horizontal) x H pixels (vertical) . In this 
embodiment, this region is segmented into three parts 
in the horizontal direction and two parts in the 
vertical direction, i.e., a total of six parts, which 
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are represented as region (0,0),..., region (2,1) 
sequentially from the upper left corner. The average 
values of R, G, and B values in each region are 
calculated, so a total of 18 numerical values are used 
5 as image feature amounts of the illustration. 

Fig. 5 is a -f low^ r ha rt . f or explaining image 
feature amount calculation processing according to the 
first embodiment. The flow of feature amount 
calculation processing will be described with reference 

10 to the flow chart shown in Fig. 5. In step S51, a 

variable k is initialized to value "0". In step S52, a 
variable j is initialized to value "0". In step S53, a 
variable i is initialized to value "0". In step S54, 
the average value of R values in a region (i,j) is 

15 substituted into a kth element d(k) of an array d. In 
a similar manner, the average value of G values and the 
average value of B values are substituted into elements 
d(k+l) and d(k+2), respectively. The method of 
calculating the average values of the R, G, and B 

20 values will be described later using the -flow chart 
shown in Fig. 6. 

In step S55, the variable k is incremented by 
three. In step S56, the variable i is incremented by 
one. In step S57, the variable i is compared with 

25 value "2". If the variable i is larger than "2", the 

flow advances to step S58. Otherwise, the flow returns 
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to step S54 . In step S58, the variable j is 
incremented by one. In step S59, the variable j is 
compared with value "1". If the variable j is larger 
than "1", the processing is ended. Otherwise, the flow 
5 returns to step S53. 

When the above processing is ended, the image 
feature amounts of the illustration are stored in an 
array d() having 18 elements. In this case, to 
£3 calculate the feature amounts, the image is segmented 

^ 10 into six rectangular regions having the same area. 

N However, the segmentation method is not limited to this. 

Q 

Ifl For example, the shape of a region is not limited to a 

p 

» rectangular shape and may be more complex. The number 

of segments is not limited to six. When the number of 
ifh 15 segments is increased/decreased, the number of elements 

of the array for storing the feature amounts is not 18 
but increases/decreases in accordance with the number 
of segments, so the values to be used for determination 
in steps S57 and S59 also change. 
J$ 20 Fig. 6 is a - flow c h art for explaining a method of 

calculating the average values of the R, G, and B 
values in each region. Assume that the image data is 
stored in three arrays R(X,Y), G(X,Y), and B(X,Y). In 
this case, 0 < X < W, and 0 < Y < H, and the origin 
25 (0,0) is set at the upper left corner of the image. In 
the following flow, the average densities in partial 
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regions of XO < X < XI and YO < Y < Yl are calculated, 
and the average values of R, G, and B values are 
substituted into variables DR, DG, and DB, respectively. 
In step S54, when the image is equally segmented 
5 into 3x2, as shown in Fig. 4, the values XO, XI, YO, 
and Yl in the region (i,j) correspond to 
XO = W x i/3, XI = W x (i + l)/3 
YO = H x j/2, Yl = H x (j + l)/2 
Hence, after the constants XO, XI, YO, and Yl are 

10 initialized to the above values, the-. flow ohart shown 
in Fig. 6 is executed. 

First, in step S61, the variables DR, DG, and DB 
are initialized to value "0". In step S62, the 
variable Y is initialized to the value YO . In step S63, 

15 the variable X is initialized to the value XO . In step 
S64, the value R(X,Y) is added to the variable DR. In 
a similar manner, the values G(X,Y) and B(X,Y) are 
added to the variables DG and DB, respectively. In 
step S65, the variable X is incremented by one. In 

20 step S66, the variable X is compared to the value XI. 

If the two values equal, the flow advances to step S67; 
otherwise, the flow returns to step S64. In step S67, 
the variable Y is incremented by one. In step S68, the 
variable Y is compared to the value Yl. If the two 

25 values equal, the flow advances to step S69; otherwise, 
the flow returns to step S63. By repeating steps S63 
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to S68, the sums of color component values 
corresponding to all pixels in the region (i,j) are 
obtained as the values DR, DG, and DB for the R, G, and 
B color components. 

In step S69, each of the values of variables DR, 
DG, and DB is divided by (XI - XO) X (Yl - YO), which 
is the total number of pixels in the region. More 
specifically, the values DR, DG, and DB are average 
densities obtained by dividing the sums of pixel 
densities in the region by the number of pixels. 
<Description of Step S23> 

In step S23, similar image search is executed on 
the basis of the above image feature amounts. The hard 
disk device 106 stores N image data, and the feature 
amounts of each image are calculated in advance in' 
accordance with the same procedure as described above 
and stored. The image data can be stored using a 
standard file format such as JPEG or Flashpix or stored 
in a so-called RDBMS (Relational DataBase Management 
System) using a unique format. Assume that image 
feature amounts are stored in a two-dimensional array 
D(n,j) having a size of N x 18 (n is an image number, 
and j is a region/color component number; in this 
embodiment, 0 < n < N, and 0 < j < 18) . The feature 
amounts of a drawn image are stored in the array d(j) 
(j is a region/color component number; 0 < j < 18). In 




this embodiment, 0-17 are allocated to the region/color 

numbers in order such as R, G and B of region (0,0), R, 

G and B of region (1,0)-" R, G and B of region (2,1). 

An inter-image distance S(n) between a presented 

5 image and the nth image is defined by 
S(n) = £(D(n,i)-d(i)) 2 

i 

It is determined that the similarity is high as 
the inter-image distance becomes small. First, the 
*f inter-image distance S (n) (0 < n < N) between the 

^ 10 presented image and each of all the N images is 

.Z? calculated, and then M ( 0 < M < N ) images are selected 

-?j sequentially in ascending order of the inter-image 

Q 

5 distances S(n), thereby executing similar image search. 

H= The processing of calculating the inter-image distance 

m 15 S(n) and the processing of selecting M images will be 

|3 described below with reference to the flow charts in 

Figs. 7 and 8, respectively. 

Referring to Fig. 7, in step S71, a variable n 
is initialized to value "0". In step S72, the variable 
20 i and inter-image distance S (n) are initialized to 
value "0". 

In step S73, it is determined whether the current 
region should be regarded as an "important region". As 
described above, the "entire image" or "central 
25 portion" is set as an important region using the 

setting menu 39. When the "central portion" is set as 
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an important region, the center of the image 
corresponds to two regions (1,0) and (1,1) because the 
image data is segmented into six regions in extracting 
the feature amounts, as shown in Fig. 4, in this 
5 embodiment. These correspond to region/color component 
numbers 3 to 5 and 12 to 14. Hence, in step S73, when 
i = 3 and i = 12, it is determined that the current 
region is an important region. When i = 3 or i = 12, 
W the flow branches to step S74; otherwise, the flow 

12 1° branches to step S75. On the other hand, when the 

^ "entire image" is designated as an important region, 

O 

10 the flow always branches to step S74. 

b In step S74, a value "100" is set to a variable A 

representing a weight. In step S75, a value "25" is 
m 15 set to the variable A. In step S76, the square of the 

JESS. 

p. product of A/100 and the difference between D(n,i) and 

d(i) is added to the value S (n) . As a result, the 
similarity in the important region is multiplied by a 
weight 100/100, and the similarity in an unimportant 

20 region is multiplied by a weight 25/100, thereby 

reflecting the contents set by the setting menu 39. 

In step S77, the variable i is incremented by 
three. In step S78, the variable i is compared with 18. 
If the two values equal, the flow advances to step S79; 

25 otherwise, the flow returns to step S73. In step S79, 
the variable n is incremented by one. In step S710, 
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the variable n is compared with the value N. If the 
two values equal, the processing is ended; otherwise, 
the flow returns to step S72. 

The inter-image distance between the presented 
5 image and each of all the stored images is stored in 
the array S(n) calculated in the above way. The 
inter-image distance is calculated with a weight 
corresponding to the "important region" or "unimportant 

Q region", as described above. Next, the processing of 

sO 

10 selecting M images sequentially in ascending order of 

Q 

% 4 inter-image distances and storing their image numbers 

10 in an array T() will be described with reference to 

Q 

a Fig. 8. 

E ts 

jKSSB 

L In step S81, the variable j is initialized to 

15 value "0". In step S82, the variable i is initialized 
to value "0". In step S83, a variable min is 
initialized to value "0", and a variable L is 
'initialized to a sufficiently large value. In step S84, 
the value S(i) is compared to the value L. If S(i) < L, 
20 the flow advances to step S85; otherwise, the flow 
advances to step S86. 

In step S85, the value i is substituted into the 
variable min and the value S(i) is substituted into the 
value L. In step S86, the variable i is incremented by 
25 one. In step S87, the variable i is compared with the 
value N. If the two values equal, the flow advances to 
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step S88; otherwise, the flow returns to step S83. In 
step S88, the value min is substituted into the array 
T(j). In step S89, a sufficiently large value is 
substituted into the value S (min) . In step S810, the 
5 variable j is incremented by one. In step S811, the 
variable j is compared with the value M. If the two 
values equal, the processing is ended; otherwise, the 
flow returns to step S82. With the above processing, 
the image numbers are stored in the array T(j) (0 < j < 
10 M) in descending order of similarities to the presented 
image . 

<Description of Step S24> 

Fig. 9 is a view showing an operation window 
displayed on the display section 103 in step S24. The 

15 thumbnail image (reduced image) of the presented 
illustration is displayed in a region 91. The 
thumbnail images (reduced images) of similar images 
searched by the above processing are displayed in 
regions 92a to 92h. An image corresponding to an image 

20 number for the highest similarity, i.e., an image 

number stored in the array T(0) is displayed in the 
region 92a, an image corresponding to the array T ( 1 ) is 
displayed in the region 92b, .... An image having the 
lowest similarity h in these images is displayed in the 

25 region 29h. As a thumbnail image, image data stored in 
the hard disk can be decoded, reduced, and displayed on 
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the window. It there is low-resolution image data for 
an icon, like Flashpix as a standard image format, that 
image data can be decoded and displayed. When a button 
93 is pressed, the next candidates, i.e., images 
5 corresponding to arrays T{8) to T(15) are displayed in 
the regions 92a to 92h as thumbnail images. This can 
be repeated up to T(M-l). When the thumbnail image (or 
icon) of a candidate image is designated, details of a 
corresponding image (original image) are displayed. 

10 When a button 94 is pressed, step S24 is ended. 

When the "central portion" is selected by the 
list button 39 shown in Fig. 3, similar search with a 
weight on the central portion of image data can be 
executed. When the "entire image" is selected, normal 

15 similar image search can be executed. 

In this embodiment, as values set to the variable 
A, 100 is used as the value on the weighted side, and 
25 is used on the other side. However, the present 
invention is not limited to this. Additionally, in 

20 this embodiment, two weight values "100" and "25" are 

used. However, more than two weight values may be used. 
In this embodiment, an image is segmented into 
1% -&cx regions, and two regions are defined as the central 
portion of the image. However, the present invention 

25 is not limited to this, and the central portion can be 
set in accordance with the number of segments or 
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segmentation method . 

For example, consider -se.arfrh^£^ra a photographic 
image database. Many photographic data generally have 
important "objects" at the central portions of the 
images. Hence, even when the user does not select the 
"central portion" by the button 38 shown in Fig. 3, 
processing with "importance on the center" may be 
.executed in the search apparatus. That is, the mode of 
"importance on the center" may be set as a default 



^ 10 state, 



[ Second Embodiment ] 

In the first embodiment, the region for 
calculation with a weight is permanently fixed. More 

15 specifically, when "importance on the center" is 

selected, weight "100" is set for the regions (1,0) and 
(1,1), and weight "25" is set for the remaining regions. 
However, the regions can be designated by the user. In 
the second embodiment, such a case will be described. 

20 As a user interface in this case, the set button 

38 shown in Fig. 3 is changed to a region designation 
button 101, as shown in Fig. 10. When the region 
designation button 101 is pressed, a rectangular region 
102 is displayed in a drawing region 31, as shown in 

25 Fig. 10. The position or size of this rectangular 

region 102 can be changed/designated by the user using 
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a pointing device such as a mouse. 

In inter-image distance calculation processing, 
when the rectangular region* 102 is designated, it is 
determined in determination processing in step S73 of 
the £luw chai.4 * shown in Fig. 7 whether the segmented 
region is included in the designated rectangular region 
102. Thus, the user can designate a rectangular region 
having an arbitrary position and size as an "important 
region" . 

The region designated by the user is not limited 
to a rectangular region. In addition, the size of the 
rectangular region is not limited to the size of a 
segmented region in calculating the image feature 
amount. In this case, however, a segmented region may 
be partially included in the designated region. In 
such a case, control is performed to change the value 
of a variable A in accordance with the state of each 
segmented region (the ratio of the region belonging to 
the rectangular region to each segmented region) , 
though the flow branches into two states upon 
determination processing in step S73 shown in Fig. 7. 

For example, steps S73 to S75 in Fig. 7 are 
replaced with steps S720 to S723 in Fig. 11 to 
determine the weight A. More specifically, first, it 
is determined in step S720 whether a region is 
designated by the user. If NO in step S720, the flow 



advances to step S723 to set 100 to the value A and 
then to step S76. If YES in step S720, the flow 
advances from step S720 to S721 to calculate the ratio 
of the region designated by the user to the segmented 
5 region as P% . In step S722, (75 x P/100) + 25 is 

calculated, the obtained value is set to the value A, 
and the flow advances to step S76. 

The layout of the operation window described in 
the above embodiments is not limited. Additionally, 
10 although a mouse has been exemplified above as a device 
used by the user to draw a sketch image, the present 
invention is not limited to this, and a pen tablet or 
s touch panel may be used. 

M, As described above, according to the first and 

15 second embodiments, the user draws an image similar to 
the desired image on the computer operation window, the 
database system segments the image into a plurality of 
regions, extracts the feature amounts in units of 
regions, and executes similar image search on the basis 
20 of the feature amounts. When similar image search is 

executed with a weight on, e.g., the central portion of 
the image or a region designated by the user in the 
plurality of segmented regions, more complex and 
advanced search is possible. Thus, an image oear - ch/ 
25 interface capable of executing saarchv &hile accurately 
reflecting the user's intention can be provided. 
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Instead of remembering the entire image, the user 
sometimes partially remembers image data as, e.g., an 
"image having such a thing at a certain portion". In 
this case, by searching for the image while placing 
5 importance on the partial region, the well-remembered 
portion can be especially "thoroughly" searched. This 
is quite effective in similar image search using a 
manually drawn image as a search source image. 

10 [Third Embodiment] 

The third embodiment will be described next. In 
the above-described first and second embodiments, the 
weight used for similarity calculation is changed in 
accordance with the position in an image. In the third 

15 embodiment to be described below, the weight used for 
similarity calculation is changed in accordance with 
contents represented by feature amounts. 

The arrangement of an image search apparatus and 
the schematic flow of image search processing in the 

20 third embodiment are the same as in the first 

embodiment (Figs. 1 and 2). In the third embodiment, 
however, similar image search processing in step S23 is 
different from the first and second embodiments. The 
difference from the first and second embodiments will 

25 be mainly described below. 

In step S21 shown in Fig. 2, a window shown in 
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Fig- 12 is displayed, and the search source image can 
be drawn using the same drawing tools as in the first 
embodiment. Fig. 12 is a view showing the operation 
window displayed on a display section 103 in the third 
embodiment. This window is different from that of the 
first embodiment (Fig. 3) in a setting menu (390) 
displayed when a set button 38 is clicked. 

As in the first embodiment, before pressing a 
search execution button 37, a condition for similar 
image search from step S22 can be changed by operating 
the set button 38. For similarity calculation in 
similar image search, "importance on luminance" or 
"importance on color differences" can be selected by 
pressing the set button 38. When the set button 38 is 
pressed, the setting menu 390 is displayed, so the user 
can select one of three conditions "importance on 
luminance", "importance on color differences", and "NO 
(importance) " . 

In step S22 shown in Fig. 2, the same image 
feature amount calculation as in the first embodiment 
(Figs. 5 and 6) is executed, and the flow advances to 
step S23. Similar image search is executed in 
accordance with the processing procedure shown in 
Fig. 13. 

Before executing the flow chart in Fig. 13, 
variables A and B are set in accordance with the 



condition set by «^te> shown in Fig. 12. For "NO", A = B 
= 50 is set. When "importance on luminance" is 
selected, for example, A = 75 and B = 25 are set. When 
"importance on color differences" is selected, for 
example, A = 25 and B = 75 are set. The values A and B 
are used for inter-image distance calculation (to be 
described later) using feature amounts. Calculation 
with "importance on luminance" and that with 
"importance on color differences" are switched 
depending on the values A and B. 

When the values A and B are set in the above way, 
in step S171, the feature amounts of the key image data 
are converted into a color space represented by the 
luminance and color differences. In this embodiment, 
the average values of R, G, and B values are converted 
into Y, Cb, and Cr values. This conversion is - 
expressed by, e.g., 

Y = 0.299R + 0.587 + 0.114B 
Cb = -0.1687R - 0.3323G + 0 . 5B + 128 
Cr = 0.5R - 0.4187G - 0.0813B + 128 
At this time, to save *te»memory, the Y, Cb, and 
Cr values may be stored in the area where the average 
values of R, G, and B values have been stored. In step 

5172, a variable n is initialed to value "0". In step 

5173, the feature amounts of the nth image data stored 
in the image database are converted into Y, Cb, and Cr 



values, as in step S171. In step S174, variables i and 
S (n) are initialized to value "0". 

In step S175, the square of the product of A/100 
or B/100 and the difference between D(n,i) and d(i) is 
5 added to the value S(n). In this example, the 

luminance value (Y) is stored in D(n,i) and D(i), the 
color difference value (Cb) is stored in D(n,i+1) and 
d(i+l), and the color difference value (Cr) is stored 
0 in D(n,i+2) and d(i+2). Hence, (D(n,i) - d(i)) is 

>4 10 multiplied by A/100, and (D(n,i+1) - d(i + l)) and 

%1 (D(n,i+2) - d(i+2)) are multiplied by B/100. 

f Fa 

ifl In step S176, the variable i is incremented by 

s~ three. In step S177, the variable i is compared with 

18- If the two values equal, the flow advances to step 
|2 15 S178; otherwise, the flow returns to step S175. In 

y step S178, the variable n is incremented by one. In 

step S179, the variable n is compared with the value N. 
If the two values equal, the processing is ended; 
otherwise, the flow returns to step S173. 
20 The inter-image distance between the presented 

image and each of all the stored images is stored in 
the array S(n) calculated in the above way. The 
inter-image distance is calculated with a weight 
corresponding to the "importance on luminance " or 
25 "importance on color differences ", as described above. 
Next, the processing of selecting M images sequentially 
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in ascending order of inter-image distances and storing 
their image numbers in an array T ( ) is executed. This 
processing is the same as described above with 
reference to Fig. 8. 
5 With the above processing, when the "importance 

on luminance " is selected by the setting menu 390 
shown in Fig. 12, similar search with a weight on 

r 

"luminance" can be executed. When the "importance on 
color differences" is selected, similar image search 
10 with a weight on "color difference" can be executed. 
In this embodiment, as values set to the 
variables A and B, 75 is used as the value on the 
important side, and 25 is used on the other side. 
However, the present invention is not limited to this. 

15 

[ Fourth Embodiment ] 

In the third embodiment, the values A and B used 

for calculation with a weight are permanently fixed. 

In the fourth embodiment, values A and B can be 
20 designated by the user. 

In this case, instead of the setting menu 390 

shown in Fig. 12, a scroll bar is used as a user 

interface, as shown in Fig. 14. Control is executed 

such that when a mark 201 is at the left end, 
25 calculation is performed with a weight on only the 

"luminance", when the mark 201 is at right end, 
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calculation is performed with a weight on only the 
"color difference", and when the mark 201 is at the 
center, calculation is performed with the same weight 
on the "luminance" and "color difference". More 
5 specifically, in the fourth embodiment, the values A 
and B for weighted similar image search processing 
(Fig. 13) described in the third embodiment are 
controlled within the range of 0 < A < 100 and 0 < B < 
□ 100. For example, A = 100 and B = 0 at the left end, A 

•4 10 =0 and B = 100 at the right end, and A = B = 50 at the 

intermediate position. 

In the third and fourth embodiments, as the 
feature amounts of image data, the average values of 
the R, G, and B values are temporarily calculated and 
15 then converted into Y, Cb, and Cr values. However, the 
present invention is not limited to this, and the Y, Cb, 
and Cr values may be calculated from the beginning. 

In addition, the layout of the operation window 
operated by the user is not limited to that of the 
20 above embodiments. 

Further, although a mouse has been exemplified 
above as a device used by the user to draw a sketch 
image, the present invention is not limited to this, 
and a pen tablet or touch panel may be used. 
25 As described above, in the third and fourth 

embodiments, a sketch pad is prepared on the computer 
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operation window, the user draws an image similar to 
the desired image on the window, and the database 
system extracts the feature amounts from the image and 
executes . similar image search on the basis of the 
feature amounts. 

When similar image search is executed by 
generating luminance information and color difference 
information from the feature amounts of the image data 
and placing importance on the luminance or color 
difference, more complex search is possible. Thus, an 
image search interface capable of executing search 
while accurately reflecting the user's intention can be 
provided . 

As described above, according to the third and 
fourth embodiments, for example, when "importance on 
luminance" is set, search can be executed mainly in 
consideration of "bright" or "dark" but not the 
chrominance information (color differences) . For this 
reason, the ambiguity for image colors remembered by 
the user can be compensated, and effective image search 
can be executed. In some cases, the user remembers not 
"colors" but "shapes" well. In this case as well, a 
satisfactory result can be obtained by search with 
"importance on luminance". 

According to the above embodiment, for example, 
when settings of the scroll bar shown in Fig. 14 are 
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changed in various ways, many search conditions can be 
generated from a drawn image. Hence, the user need not 
redraw the key search image many times. 

The present invention can be applied to a system 
5 constituted by a plurality of devices (e.g., a host 

computer, interface device, reader, and printer) or an 
apparatus constituted by a single device (e.g., a 
copying machine or facsimile apparatus) . 
D The object of the present invention can also be 

N! 10 achieved when a storage medium (or recording medium) 

a 

%j storing software program codes for realizing the 

?^ 

If! functions of the above-described embodiments is supplied 

s to a system or apparatus, and the computer (or a CPU or 

pea 

ijL an MPU) of the system or apparatus reads out and 

m 15 executes the program codes stored in the storage medium. 

In this case, the program codes read out from the 

H E 

storage medium realize the functions of the 
above-described embodiments by themselves, and the 
storage medium storing the program codes constitutes the 

20 present invention. The functions of the above-described 
embodiments are also realized not only when the readout 
program codes are executed by the computer but also when 
the OS (Operating System) running on the computer 
performs part or all of actual processing on the basis 

25 of the instructions of the program codes. 

The functions of the above-described embodiments 



- 29 - 



are also realized when the program codes read out .from 
the storage medium are written in the memory of a 
function expansion card inserted into the computer or a 
function expansion unit connected to the computer, and 
the CPU of the function expansion card or function 
expansion unit performs part or all of actual processing 
on the basis of the instructions of the program codes. 

As many apparently widely different embodiments of 
the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific 
embodiments thereof except as defined in the claims. 
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